Trim prescriptive guidance from the Freebuff prompt harness - #1276
Trim prescriptive guidance from the Freebuff prompt harness#1276thomasboom wants to merge 3 commits into
Conversation
Three layers of guidance were shipping on every model turn, eating into both latency and per-task cost without buying any model behavior: 1. base3 system prompt + CLI appendix dropped from ~40 lines of prescriptive bullets to a 3-line system prompt and a one-line identity string. The destructive-command safety rail stays; the "match conventions / verify libs / use write_todos" bullets were product-guidance dressed up as engineering judgement. 2. Eight tool descriptions trimmed to what the model actually needs to call the tool: code-search lost a 100-line ripgrep flag essay plus result-limiting notes; write-todos lost a 4-bullet manual; suggest-followups lost its "Good suggestions include" / "Don't include" lists; web-search / list-directory / glob lost their Purpose/Use-cases essays; run-terminal-command lost the 6-bullet "Stick to these use cases" and the 9-bullet <commit_analysis> checklist (the destructive-command rail is already in base3's system prompt). ask-user lost its redundant options list but kept the IMPORTANT no-Custom-options rule. Schema fields untouched - the model still knows each parameter. 3. Deleted the unused getToolsInstructions / fullToolList / getShortToolInstructions exports in packages/agent-runtime/src/tools/prompts.ts (the [CRITICAL] Formatting Requirements, DO NOT narrate tool names, AS LITTLE TEXT AS POSSIBLE optimization) and the unused knowledgeFilesPrompt in system-prompt/prompts.ts. None had a live caller; the AI SDK reads each tool's description directly via getToolSet. Per-turn hot cost: ~4,400 tokens saved on every model call (8 tool descriptions: ~13 KB / ~3,800 tokens; base3 system prompt: ~2 KB / ~625 tokens, cache-warm). On a 20-step task that's ~88k tokens not billed. base2, base-chat, and base-deep untouched - different surfaces. base2 is the legacy/free-fallback harness and the eval control arm; base-chat is the freebuff.com/chat surface; base-deep is the deprecated deep-reasoning path.
|
The dead-code removal is clean and verifiable: The rest is a much bigger claim than the diff can support. Prompt wording for a coding agent (especially the base3 harness that's presumably the primary path) is usually tuned against eval suites, not just token-cost math. The PR asserts "buying no model behavior" for things like "match conventions," "verify libs before using them," and "use write_todos," but offers no eval numbers, only a token-savings estimate. That's a real product-behavior change dressed as a cleanup. More concretely risky: Also telling: the test changes ( Split this into (1) the dead-export removal, which is a clean, small, verifiable PR, and (2) the prompt-content trims, which need eval evidence or at minimum a narrower, one-tool-at-a-time PR so a maintainer can judge each trim on its own. |
codebuff-team
added
bot:triaged
2 participants
Three layers of guidance were shipping on every model turn, eating into both latency and per-task cost without buying any model behavior, while boating the context window and therefore making model performance worse:
base3 system prompt + CLI appendix dropped from ~40 lines of prescriptive bullets to a 3-line system prompt and a one-line identity string. The destructive-command safety rail stays; the "match conventions / verify libs / use write_todos" bullets were product-guidance dressed up as engineering judgement.
Eight tool descriptions trimmed to what the model actually needs to call the tool: code-search lost a 100-line ripgrep flag essay plus result-limiting notes; write-todos lost a 4-bullet manual; suggest-followups lost its "Good suggestions include" / "Don't include" lists; web-search / list-directory / glob lost their Purpose/Use-cases essays; run-terminal-command lost the 6-bullet "Stick to these use cases" and the 9-bullet <commit_analysis> checklist (the destructive-command rail is already in base3's system prompt). ask-user lost its redundant options list but kept the IMPORTANT no-Custom-options rule. Schema fields untouched - the model still knows each parameter.
Deleted the unused getToolsInstructions / fullToolList / getShortToolInstructions exports in packages/agent-runtime/src/tools/prompts.ts (the [CRITICAL] Formatting Requirements, DO NOT narrate tool names, AS LITTLE TEXT AS POSSIBLE optimization) and the unused knowledgeFilesPrompt in system-prompt/prompts.ts. None had a live caller; the AI SDK reads each tool's description directly via getToolSet.
Per-turn hot cost: ~4,400 tokens saved on every model call (8 tool descriptions: ~13 KB / ~3,800 tokens; base3 system prompt: ~2 KB / ~625 tokens, cache-warm). On a 20-step task that's ~88k tokens not billed.
base2, base-chat, and base-deep untouched - different surfaces. base2 is the legacy/free-fallback harness and the eval control arm; base-chat is the freebuff.com/chat surface; base-deep is the deprecated deep-reasoning path.